Papers with multi-step tasks
You Only Look at Screens: Multimodal Chain-of-Action Agents (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to creating autonomous graphical user interfaces rely on external tools and application-specific APIs to interpret the environment. |
| Approach: | They propose a multimodal solution that directly interacts with the user interface without environment parsing. |
| Outcome: | The proposed solution bypasses environment parsing and reliance on application-dependent APIs. |
PRompt Optimization in Multi-Step Tasks (PROMST): Integrating Human Feedback and Heuristic-based Sampling (2024.emnlp-main)
Copied to clipboard
| Challenge: | Prompt optimization aims to find the best prompt to a large language model (LLM) for a given task. |
| Approach: | They propose a method to optimize prompts for LLM-driven multi-step tasks using a human-designed feedback rule. |
| Outcome: | The proposed method outperforms human-engineered prompts and several other prompt optimization methods on 11 representative multi-step tasks. |
PEAR: Planner-Executor Agent Robustness Benchmark (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing studies examine isolated attack surfaces or specific scenarios, leaving a lack of holistic understanding of MAS vulnerabilities. |
| Approach: | They propose a benchmark to evaluate the utility and vulnerability of planner–executor MAS. |
| Outcome: | The proposed benchmark evaluates planner–executor MAS on a widely adopted design. |
Implicit Reasoning in Transformers is Reasoning through Shortcuts (2025.findings-acl)
Copied to clipboard
| Challenge: | Language models can perform step-by-step reasoning and achieve high accuracy in both in-domain and out-of-domain tests via implicit reasoning. |
| Approach: | They train GPT-2 from scratch on a curated multi-step mathematical reasoning dataset and conduct analytical experiments to investigate how language models perform implicit reasoning in multi- step tasks. |
| Outcome: | The proposed model performs better on multi-step tasks than the explicit reasoning model. |
Idola Tribus of AI: Large Language Models tend to perceive order where none exists (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a tendency of large language models to generate absurd patterns is observed . authors say this is a limitation of the models' ability to perform complex tasks . |
| Approach: | We present a tendency of large language models to generate absurd patterns . authors conducted an experiment to evaluate logical consistency and self-coherence of LLMs . |
| Outcome: | a recent study shows that large language models generate absurd patterns despite their inadequacy . the model over-recognized patterns that were inconsistent with the given numbers, the study finds . |
WindowsWorld: A Process-Centric Benchmark of Autonomous GUI Agents in Professional Cross-Application Environments (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing GUI agents perform poorly on multi-application tasks, stalling at early sub-goals. |
| Approach: | They propose to assess GUI Agents on complex multi-step tasks that mirror real-world professions. |
| Outcome: | The proposed benchmark contains 181 tasks with an average of 5.0 sub-goals across 17 common desktop applications, of which 78% are inherently multi-application. |
LoopTool: Closing the Data–Training Loop for Robust LLM Tool Calls (2026.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are powerful tools for multi-step tasks, but static data pipelines hinder tool learning and cause noisy labels to persist. |
| Approach: | They propose a fully automated, model-aware data evolution framework that tightly integrates data synthesis and model training. |
| Outcome: | Experiments show that LoopTool-8B significantly surpasses its 32B data generator and achieves new state-of-the-art results on the BFCL-v3 and ACEBench benchmarks for its scale. |
a1: Steep Test-time Scaling Law via Environment Augmented Generation (2026.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have made remarkable advances in reasoning, yet continue to struggle with hallucinations, logical errors, and inability to self-correct during complex multi-step tasks. |
| Approach: | They propose a framework that enhances LLM reasoning through real-time environmental feedback validating each reasoning step, dynamic branch exploration for investigating alternative solution paths when faced with errors, and experience-based learning from successful reasoning trajectories. |
| Outcome: | The proposed model outperforms comparable models by 24.4 percentage points across benchmarks while outperforming comparable models. |
LLM Agents Making Agent Tools (2025.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) can perform multi-step tasks by dynamically utilising external software components. |
| Approach: | They propose an agentic framework that autonomously transforms papers with code into LLM-compatible tools. |
| Outcome: | The proposed framework outperforms current state-of-the-art software engineering agents in 80% of tasks and is openly available on GitHub. |
ConsistencyChecker: Tree-based Evaluation of LLM Generalization Capabilities (2025.acl-long)
Copied to clipboard
| Challenge: | Traditional self-consistency methods fail to capture subtle semantic errors in multi-step tasks. |
| Approach: | They propose a tree-based evaluation framework that measures LLMs’ ability to preserve semantic consistency during reversible transformations. |
| Outcome: | The proposed framework measures generalization abilities across models from 1.5B to 72B and can be used to benchmark LLMs without constructing new datasets. |
Action Boundary Blindness: When LLM Agents Cannot Tell Where One Action Ends and Another Begins (2026.acl-long)
Copied to clipboard
| Challenge: | Large language model agents exhibit action boundary blindness, granularity confusion, scope creep and boundary ambiguity . Explicit boundary prompting improves ABS by 0.08–0.13 across all models . |
| Approach: | They propose four automatic metrics that require no human annotation to detect boundary blindness . they propose to use a multi-label attribution framework to validate the models . |
| Outcome: | Experiments with seven large language model agents show that the best model achieves only 0.424 ABS . Explicit Boundary Prompting improves ABS by 0.08–0.13 across all models . |
TIME: Temporally Intelligent Meta-reasoning Engine for Context-Triggered Explicit Reasoning (2026.findings-acl)
Copied to clipboard
| Challenge: | Reasoning-oriented language models expose explicit reasoning as a long, front-loaded chain of “thinking” tokens before the main output, either always enabled or externally toggled at inference time. |
| Approach: | They introduce a behavioral alignment framework that learns explicit reasoning as a context-triggered control policy rather than a fixed response mode. |
| Outcome: | The proposed framework improves TIMEBench scores over the base model in thinking and no-thinking modes while keeping output compact. |
ReasMark: A Robust Watermark for Attributing LLM Reasoning Under Knowledge Distillation Attacks (2026.acl-long)
Copied to clipboard
Peizhuo Lv, Ruihua Zhou, Yunpeng Li, Ruigang Liang, Xingshuo Han, XiaoFeng Wang, Wei Dong, Yuling Liu
| Challenge: | Existing reasoning-enhanced large language models fail to provide reliable attribution of reasoning behavior once it is transferred through knowledge distillation. |
| Approach: | They propose to embed a reasoning-length gap in a model by querying a target domain and training a local student to imitate its outputs. |
| Outcome: | et al. show that ReasMark outperforms baselines while preserving task utility. |